Summary of Changes

Hello @jingyu-ml, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands vLLM's quantization capabilities by integrating full support for ModelOPT FP8 quantized models, with a particular focus on Qwen models. It introduces new quantization methods for linear and Mixture-of-Experts (MoE) layers, refines the weight loading process to accommodate ModelOPT's specific checkpoint formats and scale handling, and includes model-specific adaptations to ensure seamless loading and execution of these quantized models. The changes aim to broaden the range of supported models and improve loading efficiency for quantized architectures.

Highlights

• ModelOPT FP8 Quantization Support: Added comprehensive support for ModelOPT FP8 quantization, enabling vLLM to load and run models quantized with this method. This includes identifying ModelOPT FP8 from Hugging Face configurations and handling its specific data formats.
• Mixture-of-Experts (MoE) Layer Quantization: Introduced ModelOptFp8MoEMethod to specifically handle FP8 quantization for MoE layers. This new method manages the creation of FP8 weights and scales, processes scales after loading (e.g., combining w1 and w3 scales), and applies the quantization during the forward pass.
• Flexible Weight Loading Enhancements: Improved the weight loading mechanism across various models to dynamically adapt to different ModelOPT weight scale patterns (e.g., weight_scale vs. weight_scale_2) and to correctly handle complex expert scale broadcasting from single values to multiple experts. This includes updates to llama4, mllama4, qwen2, and qwen3_moe.
• Performance Optimizations for MoE: Introduced supports_moe_loading flags to relevant weight loaders (FusedMoE, experts_int8, moe_wna16) to explicitly mark them as MoE-aware. This optimization helps avoid expensive runtime reflection during model loading, improving efficiency for MoE architectures.
• KV Cache Scale Remapping: Enhanced the maybe_remap_kv_scale_name utility to correctly remap KV cache scale names, including support for qkv_proj parameters, ensuring accurate loading of attention-related scales in quantized models.

Code Review

This pull request adds support for Qwen models with FP8 quantization using ModelOPT. The changes span several files, including configuration, model layers, and weight loading utilities. Key additions include logic to parse ModelOPT-specific quantization configurations and a new ModelOptFp8MoEMethod for handling MoE layers. Overall, the changes are well-structured to support the new quantization format.